Skip to content

feat(automation): durable run history with failure reasons (run observability) - #2581

Merged
os-zhuang merged 1 commit into
mainfrom
feat/automation-run-history
Jul 4, 2026
Merged

os-zhuang merged 1 commit into
mainfrom
feat/automation-run-history

Conversation

@os-zhuang

Copy link
Copy Markdown
Contributor

Why

Automation runs were observable only in memory. The engine kept the last N runs in a ring buffer, so after a process restart (or once the buffer evicted the entry) you could not answer "did this flow run, and why did it fail?" — and a failed run surfaced no reason at all. This was the single biggest silent-trust gap for anyone authoring automations: a flow could stop firing or start failing with nothing durable to inspect.

This is the #1 UX recommendation from the online-automation dogfood: run observability — durable, queryable runs with status + failure reason.

What

sys_automation_run — previously the ADR-0019 store for live suspended runs only — becomes a durable run-history table.

  • engine — recordLog mirrors every terminal run (completed / failed) to the store via recordTerminal (fire-and-forget). listRuns() merges durable history with the in-memory buffer (in-memory wins on id, newest-first) so the Studio "Runs" surface shows runs that predate the current process.
  • store — recordTerminal / listHistory on both SuspendedRunStore impls. Terminal history rows use a run_-prefixed id, disjoint from live suspended runs (raw runId, status: 'paused', deleted on completion), so the suspend save/load/delete/list path is untouched and resume sweeps (list() filters status: 'paused') never see history rows.
  • schema — finished_at / duration_ms / error (all optional) + the status enum gains running / completed / failed alongside paused.

Safe & additive

  • Distinct id namespace ⇒ zero collision with the ADR-0019 suspend/resume lifecycle.
  • Persisting is best-effort + fire-and-forget — a history write/read failure is logged and swallowed, never breaking the run that produced it.
  • New fields optional; new store methods optional (in-memory/test defaults still work).

Verification

Unit — new run-history.test.ts: completed persistence, failed persistence with error, read-across-restart (fresh engine over the same store), best-effort isolation (a throwing store never breaks the run). Full suite green (231 passed), DTS typecheck clean.

Runtime, end-to-end on a clean showcase instance (the standard that caught the earlier no-op #2560 fix — a fake-service unit test alone can mask a runtime-broken change):

  • schedule-triggered showcase_scheduled_digest + 7 task-completion flows → durable completed rows with finished_at;
  • showcase_resilient_sync → a failed row carrying its try_catch failure reason;
  • a live showcase_task_follow_up paused suspended run coexisted without collision;
  • after a full process restart, the failed row — reason intact — was still queryable via /api/v1/data/sys_automation_run.

🤖 Generated with Claude Code

…vability)

Mirror every terminal automation run (completed / failed) to the
sys_automation_run table so "did it run, and why did it fail?" survives a
process restart and the in-memory ring-buffer eviction — previously runs were
observable only in memory and failures surfaced no reason.

- engine: `recordLog` fire-and-forget `recordTerminal` on terminal runs;
  `listRuns` merges durable history with the in-memory buffer (freshest wins).
- store: `recordTerminal` / `listHistory` on both SuspendedRunStore impls,
  keyed by a `run_`-prefixed id disjoint from live suspended runs so the
  ADR-0019 suspend/resume path is untouched and resume sweeps skip history.
- schema: sys_automation_run gains `finished_at` / `duration_ms` / `error`
  (all optional) and a `completed` / `failed` / `running` status alongside
  `paused`; becomes a durable run-history table, not suspend-only.
- best-effort throughout: a history write/read failure is logged and swallowed,
  never breaking the run that produced it.

Verified end-to-end on a clean showcase: completed + failed runs persisted
durable rows (failure reason intact), a live paused run coexisted without
collision, and the failed row survived a full process restart. New
run-history.test.ts covers persistence, read-across-restart, best-effort isolation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 4, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
spec Ready Ready Preview, Comment Jul 4, 2026 1:56pm

Request Review

@github-actions github-actions Bot added documentation Improvements or additions to documentation tests tooling size/m labels Jul 4, 2026
@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): packages/services.

5 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:

  • content/docs/guides/packages.mdx (via packages/services)
  • content/docs/guides/runtime-services/audit-service.mdx (via packages/services)
  • content/docs/guides/runtime-services/index.mdx (via packages/services)
  • content/docs/guides/runtime-services/settings-service.mdx (via packages/services)
  • content/docs/protocol/objectos/i18n-standard.mdx (via packages/services)

Advisory only. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs origin/main → pass the list as args.docs.

@os-zhuang
os-zhuang merged commit ffafb30 into main Jul 4, 2026
16 checks passed
@os-zhuang
os-zhuang deleted the feat/automation-run-history branch July 4, 2026 14:01
This was referenced Jul 5, 2026

This branch was successfully deployed

1 active deployment
Preview — 1d3443a7 Deployed Jul 4, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/m tests tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant